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(54) Adaptive colour plane interpolation In an electronic single-sensor colour camera 



(57) Apparatus is described for processing a digi- 
tized image signal obtained from an image sensor hav- 
ing color photosites aligned in rows and columns that 
generate at least three separate color values but only 
one color value for each photosite location, means for 
interpolating color values for each photosite location so 
that it has three different color values. The apparatus 
includes a memory for storing the digitized image signal 
and a processor operative with said storing means for 
generating an appropriate color value missing from a 
photosite location by the interpolation of an additional 
color value for such photosite locations from color val- 
ues of different colors than the missing color value at 



nearby photosite locations. The processor also includes 
structure for obtaining Laplacian second-order values 
and gradient values in at least two image directions from 
nearby photosites of the same column and row and for 
adding the Laplacian second-order values and the gra- 
dient values to define a classifier and for selecting a pre- 
ferred orientation for the interpolation of the missing 
color value based upon a classifier. Finally, a arrange- 
ment is provided for interpolating the missing color 
value from nearby multiple color values selected to 
agree with the preferred orientation. 
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Description 

r.rr«s Referee *n Related Applications 

i .cc^iiS Serial No 08/407,436 filed March 17, 1995 to James E. Adams, Jr. 

of fop Invention 

„ This invention rdates to the M of eiectronic imaging and is particular* su^e to electric stii. imaging by 
nJ^SSi* still camera having a single color sensor and memory. 

Pfl/^ rnunri of the Invention 

„ lnelectronicco.0^^^ 

green and blue. When the three color planes are corrfc.neo, *^°~J^J photography. *s is sometimes accom- 
?hese three sets of image data can be done ^number I by a pa?ern i red. green and blue filters. 

^oS^^^^ 
^eTat^ 

ima °* processins techrtq r 

ro ^e 0 «° Xs ho* red green and blue pixels can be arranged in a co.or «ter array. For a more detarfed 
description see U.S. Patent No. 3,971 ,065 to Bayer. 
GRGR 
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Rtimmarv <?f Invention 



Tneobiect ofth.inventionistopro.de an improv^^ 
This object is achie^ in apparatus for r^ess.ng a d.^ 

color values comprising: 

means for storing the digitized image signal; nanmrKSna aooropriate color value missing from a photosite 
a processor operative with said stonng means for 9*"^^ from color values of different 

,ocation by the interpofction of an addition* «£^*JJ* JS^eSor including 
. SET ^JSX^^S^^ - at .east « in.ge directions from 

50 ferred orientation. 
Advantages 
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Brief Description of the Drawings 

FIG. 1 is a block diagram of an electronic still camera employing interpolation processing according to the invention; 
FIG. 2 is a block diagram of the logic of the interpolation processing technique used in connection with the inven- 
s tion; 

FIG. 3 is a detailed block diagram of the logic of the interpolation luminance section in FIG. 2; 
FIG. 4 is a more detailed block diagram of the logic of the chrominance section in FIG. 2; and 
FIGS. 5 and 6 show logic block diagrams for the interpolation processes in accordance with the invention. 

10 Petailed Pescrlption of the Invention 

Since single-sensor electronic cameras employing color filter arrays are well known, the present description will be 
directed in particular to elements forming part of. or cooperating more directly with, apparatus and method in accord- 
ance with the present invention. Elements not specifically shown or described herein may be selected from those 
is known in the art. 

Referring initially to FIGS. 1 and 2, an electronic still camera is divided generally into an input section 2 and an inter- 
polation and recording section 4. The input section 2 includes an exposure section 10 for directing image light from a 
subject (not shown) toward an image sensor 12. Although not shown, the exposure section 10 includes conventional 
optics for directing the image light through a diaphragm, which regulates the optical aperture, and a shutter, which reg- 

20 ulates exposure time. The image sensor 12, which includes a two-dimensional array of photosites corresponding to pic- 
ture elements of the image, is a conventional charge-coupled device (CCD) using either well-known interline transfer or 
frame transfer techniques. The image sensor 12 is covered by a color filter array (CFA) 13, known as the Bayer array, 
which is described in U.S. Patent No. 3,971 ,065 and herewith incorporated by reference. In the Bayer geometry each 
color covers a photosite, or picture element (pixel), of the sensor. In particular, chrominance colors (red and blue) are 

25 interspersed among a checkerboard pattern of luminance colors (green). The image sensor 12 is exposed to image 
light so that analog image charge information is generated in respective photosites. The charge information is applied 
to an output diode 14, which converts the charge information to analog image signals corresponding to respective pic- 
ture elements. The analog image signals are applied to an A/D converter 16, which generates a digital image signal 
from the analog input signal for each picture element. The digital signals are applied to an image buffer 18, which may 

30 be a random access memory (RAM) with storage capacity for a plurality of still images. 

A control processor 20 generally controls the input section 2 of the camera by initiating and controlling exposure 
(by operation by the diaphragm and shutter (not shown) in the exposure section 10), by generating the horizontal and 
vertical clocks needed for driving the image sensor 12 and for clocking image information therefrom, and by enabling 
the A/D converter 16 in conjunction with the image buffer 18 for each signal segment relating to a picture element. (The 

35 control processor 20 would ordinarily include a microprocessor coupled with a system timing circuit.) Once a certain 
number of digital image signals have been accumulated in the image buffer 1 8, the stored signals are applied to a digital 
signal processor 22, which controls the throughput processing rate for the interpolation and recording section 4 of the 
camera. The digital signal processor 22 applies an interpolation algorithm to the digital image signals, and sends the 
interpolated signals to a conventional, removable memory card 24 via a connector 26. 

40 Since the interpolation and related processing ordinarily occurs over several steps, the intermediate products of the 
processing algorithm are stored in a processing buffer 28. (The processing buffer 28 may also be configured as part of 
the memory space of the image buffer 18.) The number of image signals needed in the image buffer 18 before digital 
processing can begin depends on the type of processing, that is, for a neighborhood interpolation to begin, a block of 
signals including at least a portion of the image signals comprising a video frame must be available. Consequently, in 

45 most circumstances, the interpolation may commence as soon as the requisite block of picture elements is present in 
the buffer 18. 

The input section 2 operates at a rate commensurate with normal operation of the camera while interpolation, 
which may consume more time, can be relatively divorced from the input rate. The exposure section 10 exposes the 
image sensor 12 to image light for a time period dependent upon exposure requirements, for example, a time period 
so between 1/1 000 second and several seconds. The image charge is then swept from the photosites in the image sensor 
12, converted to a digital format, and written into the image buffer 18. The driving signals provided by the control proc- 
essor 20 to the image sensor 12, the A/D converter 16 and the buffer 18 are accordingly generated to achieve such a 
transfer. The processing throughput rate of the interpolation and recording section 4 is determined by the speed of the 
digital signal processor 22. 

55 One desirable consequence of this architecture is that the processing algorithm employed in the interpolation and 
recording section may be selected for quality treatment of the image rather than for throughput speed. This, of course, 
can put a delay between consecutive pictures which may affect the user, depending on the time between photographic 
events. This is a problem since it is well known and understood in the field of electronic imaging that a digital still camera 
should provide a continuous shooting capability for a successive sequence of images. For this reason, the image buffer 
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of >he d«. Such l*™*»n Mude W^*^^,^ 5 ^ (apenu™ 

number of image spaces remaining, or estimated to be , b^e, 18 according to the 
The digital signal processor 22 interpolates each sWI vriec Mmage stor« Mr > ™ ^ fo||ows ^ 

interpoiataontechn^e^ 

SpTit=n5:^e^^^ 

is interpoiated at the high frequency locations . by section 36 for opti- 

the implementation shown in FIG. 2. an adapts ,nt *^°^f^^^^^ green" pixels are adap- 

mizing the performance of the system for m ^^Z^J^e^n^nZ gradient established between 
tively interpolated either horizontally, vertically or ^'™7"°^ the "missing green" pixel, 

the chrominance (red and blue) pixe. l0 ^ 10 ^ , ^ the ^ e ^l S Werpotetion method. Thedetails 
The first step *££££o3!2 f^^^^^Lo^^^.^ 
of this process are shown in block 40 of Fie. 3. i neproce^^ «y » classifier" denotes a value computed 

(Block 50).onefor the horizoma. direction andonefortheve^ 

tor the purpose of making a decision about further proce^ng of the pixel "J 0 ^^^ * the £>lacian of the 
spending color in the Bayer color filter array. ^ that one ^ ue is 

red or blue filter in the Bayer color f .Iter array. 0uma) interpolation described 

The red/blue (chroma) interpolator, Proceeds ."^ .™ ^^J^oon^^ccnv^^ 
35 above. The details of this process are shown .n Block 46 of | £^Tta pw« ^ » ^3 term . negat i V e 

« plane, depending on which 2"2^SSSo* 62) and tested for equality. In the likely event that one value is 

The two classifier values are then compared (Block ezj ana ibkbo _ «h j & ^ ^ „ tne ^,^5 

smaller than the other, the interpolation method WE*^^*^ £25 s done. 

"Cre^e^ 
example using the Bayer array. 

50 

firpen Plane Inte rpolation 

The firs, pass of the interpolation fully populates the green color plane. The Bayer color fitter array is assumed. 
Consider the following neighborhood. 
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Gx is a green pixel and Ax is either a red pixel or a blue pixel. (All Ax pixels will be the same color for the entire 
neighborhood.) For simplicity, we will use the term "chroma" to mean either red or blue. We form the following classifi- 
ers. 

15 

DH = |-A3 + 2A5 - A7| + |G4 - G6| 

DV o |-A1 + 2A5 - A9| + |G2 - G8| 

20 These classifiers are composed of Laplacian second-order terms for the chroma data and gradients for the green 
data. As such, these classifiers are sensing the high spatial frequency information present in the pixel neighborhood in 
the horizontal (DH) and vertical (DV) directions. 
We then form three predictors. 

25 G5H = (G4 + G6)/2 + (-A3 + 2A5 - A7)/4 

G5V = (G2 + G8)/2 + (-A1 + 2A5 - A9)/4 

G5A = (G2 + G4 + G6 + G8)/4 + (-A1 - A3 + 4A5 - A7 - A9)/8 

30 

These predictors are composed of arithmetic averages for the green data and appropriately scaled Laplacian sec- 
ond-order terms for the chroma data. G5H is to be used when the preferred orientation for the interpolation is in the hor- 
izontal direction within the pixel neighborhood. Similarly, G5V is to be used when the preferred orientation for the 
interpolation is the vertical direction. G5A is used when there is no clear preference for orientation for the interpolation. 
35 The complete green interpolation process may now be expressed as below. 



IF DH < DV 

THEN G5 = G5H 
ELSE IF D V < DH 

THEN G5 = G5V 
ELSE G5 = G5A 



The key to this process is that both the green and the chroma data must indicate a minimum of high spatial fre- 
quency information for a given orientation to be chosen as the preferred orientation for the interpolation. If there is a 
large amount of high spatial frequency information in either the green data or chroma data for a given orientation, it will 
so inflate the value of the corresponding classifier. This, in turn, reduces the likelihood for that orientation to be chosen as 
the preferred orientation for the interpolation. 
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In practice, the green interpolation process may 
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be simplified, as below, for computational efficiency. 



IFDH<=DV 

THEN G5 = G5H 
ELSE G5 = G5V 



image quality of the final reconstructed .mage. 
)5 onH RI..B tO-""-) '"ternolation 

jSn summlrizi) below. Consider the following neighborhood. 

so A1 G2 A3 

G4 C5G6 

A7 G8 A9 u . • «i orv< k the oDDOSite color pixel to Ax (i.e., if Ax is red then C5 

is used. (A4 is used as an ©ample.) 

A4 = (A1 + A7)/2 + (-Q1 + 2Q4 - G7)/2 

example.) 

A2 = (A1 + A3)/2 + (-G1 + 2G2 - G3)/2 

- c-..- — ~™ — '»-* 

an example.) 

A5 = (A1 + A3 + A7 + A9)/4 + (-G1 - C33 - 4G5 - Q7 - G9V4 

We describe an improvement for case 3. We form the following two classif iers. 

DN = I-Q1 + 2G5 - G9| + |A1 - A9| 
DP = |-G3 + 2G5 - G7| + |A3 - A7| 

?n?n^?diagona. (DN) and port* diagonal (DP) directon, 
so We then form three predictors. 

A5N = (A1 + A9)/2 + (-G1 + 2G5 - G9V2 
A5P = (A3 + A7)/2 + (-G3 + 2G5 - G7)/2 
A5A = (A1 + A3 + A7 + A9)/4 + (-G1-G3 + 4G5-G7-G9V4 
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negative diagonal direction within the pixel neighborhood. Similarly, ASP is to be used when the preferred orientation 
for the interpolation is the positive diagonal direction. A5A is used when there is no clear preference for orientation for 
the interpolation. 

The complete case 3 chroma interpolation process may now be expressed as below. 



IF DN < DP 

THEN A5 = A5N 

ELSE IF DP < DN 

THEN A5 = ASP 
ELSE A5 = ASA 



In this process both the green and the chroma data must indicate a minimum of high spatial frequency information 
for a given orientation to be chosen as the preferred orientation for the interpolation. If there is a large amount of high 
spatial frequency information in either the green data or chroma data for a given orientation, it will inflate the value of 
the corresponding classifier. This, in turn, reduces the likelihood for that orientation to be chosen as the preferred ori- 
entation for the interpolation. 

In practice, the case 3 chroma interpolation process may be simplified, as below, for computational efficiency. 



IFDN<=DP 

THEN AS = A5N 
ELSE AS = ASP 



This simplification is achieved by defining the negative diagonal direction as the default preferred orientation for the 
interpolation when both negative and positive diagonal classifiers are equal. The number of occurrences in a typical 
image when the negative and positive diagonal classifiers are equal is so small that this simplification generally has 
negligible impact on the image quality of the final reconstructed image. 

Other features of the invention are included below. 

The apparatus wherein, when the preferred orientation is vertical, a color site is green, and the missing color value 
is red, adjacent vertical red color values are averaged and vertical green color values including the green color site 
value provide the correction value. 

The apparatus wherein, when the preferred orientation is vertical, a color site is green, and the missing color value 
is blue, adjacent vertical blue color values are averaged and vertical green color values including the green color site 
value provide the correction value. 

The apparatus wherein, when the preferred orientation is positive diagonal, a color site is green, and the missing 
color value is red, adjacent positive diagonal red color values are averaged and positive diagonal green color values 
including the green color site value provide the correction value. 

The apparatus wherein, when the preferred orientation is positive diagonal, a color site is green, and the missing 
color value is blue, adjacent positive diagonal blue color values are averaged and positive diagonal green color values 
including the green color site value provide the correction value. 

The invention has been described in detail with particular reference to certain preferred embodiments thereof, but 
it will be understood that variations and modifications can be effected within the spirit and scope of the invention. 

PARTS LIST 

2 input section 

4 recording section 

1 0 exposure section 

12 image sensor 

13 color filter array 

14 output diode 
16 A/D converter 
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18 image buffer 

20 control processor 

22 digital signal processor 

24 removable memory card 

» 26 connector 

28 processing buffer 

30 display panel 

36 luminance section 

38 chroma section 

w 40 select best luma interpolation 

44 interpolate missing luma values 

46 select best chroma interpolation 

48 interpolate missing chroma values 

£ Sute horizontal and vertical conposiste classrfer values 

56 select default method for interpolation 
60 block 
62 block 
20 64 block 
66 block 

70 average red/blue values 
72 add green correction factor 
80 block 
25 82 block 
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means for storing the digitized image signal; nexnmfinQ an appropriate color value missing from a pho- 

preferred orientation. 

different color values comprising: 

means for storing the digitized image signal; appropriate color value missing from a pho- 

a processor operative with said stonng mw«ta genera "g an a^rop from ^ ^ of 

tosite location by the interpolation of an f^?"^ 0 ™* ^ processor including 

nearby photosites of the same column and row. e a classifl er and for 
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means for interpolating the missing color value from nearby multiple color values selected to agree with the 
preferred orientation wherein neighboring color values of the same color are averaged and corrected by neigh- 
boring color values of a different color value. 

s 3. The apparatus of claim 1 wherein, when the preferred orientation is horizontal, a color site is red, and the missing 
color value is green, adjacent horizontal green color values are averaged and horizontal red color values including 
the red color site value provide the correction value. 

4. The apparatus of claim 1 wherein, when the preferred orientation is horizontal, a color site is blue, and the missing 
10 color value is green, adjacent horizontal green color values are averaged and horizontal blue color values including 

the blue color site value provide the correction value. 

5. The apparatus of claim 1 wherein, when the preferred orientation is horizontal, a color site is green, and the missing 
color value is red, adjacent horizontal red color values are averaged and horizontal green color values including the 

15 green color site value provide the correction value. 

6. The apparatus of claim 1 wherein, when the preferred orientation is horizontal, a color site is green, and the missing 
color value is blue, adjacent horizontal blue color values are averaged and horizontal green color values including 
the green color site value provide the correction value. 

20 

7. The apparatus of claim 1 wherein, when the preferred orientation is negative diagonal, a color site is green, and the 
missing color value is red, adjacent negative diagonal red color values are averaged and negative diagonal green 
color values including the green color site value provide the correction value. 

25 8. The apparatus of claim 1 wherein, when the preferred orientation is negative diagonal, a color site is green, and the 
missing color value is blue, adjacent negative diagonal blue color values are averaged and negative diagonal green 
color values including the green color site value provide the correction value. 

9. The apparatus of claim 1 wherein, when the preferred orientation is vertical, a color site is red, and the missing 
30 color value is green, adjacent vertical green color values are averaged and vertical red color values including the 

red color site value provide the correction value. 

10. The apparatus of claim 1 wherein, when the preferred orientation is vertical, a color site is blue, and the missing 
color value is green, adjacent vertical green color values are averaged and vertical blue color values including the 

35 blue color site value provide the correction value. 
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